Heterogeneous-Branch Collaborative Learning for Dialogue Generation

With the development of deep learning, advanced dialogue generation methods usually require a greater amount computational resources. One promising approach to obtaining high-performance and lightweight model is knowledge distillation, which relies heavily on pre-trained powerful teacher. Collaborative also known as online an effective way conduct one-stage group distillation in absence well-trained large teacher model. However, previous work has severe branch homogeneity problem due same training objective independent identical sets. To alleviate this problem, we consider attributes network branches. Each learns attribute-related features based selected subset. Furthermore, propose dual group-based method, consisting positive negative further diversify different branches steadily interpretable way. The proposed significantly improves heterogeneity outperforms state-of-the-art collaborative learning two widely used open-domain datasets.